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ABSIBACT 

Faculty perceptions of simulated college student 
evaluative infcraation collected by three methods were investigated: 
open-ended questionsr group interviews # and objective questionnaire 
itess^ The objective items were from the Instructor and Course 
Evaluation System (ICES) student rating questionnaires* open-ended 
questions were: (1) comment on the strengths and weaknesses of the 
instructor 5 i2) the beneficial aspects of the course; (3) suggestions 
for course iitFrovement; and (U) comment on the grading procedures* 
The group interview process involved a pre- and post- interview 
session with the interviewer and the instructor-. Faculty rated the 
three simulated evaluation reports on their potential for accuracy, 
trustworthiness, usefulness r comprehensiveness, believability , 
inter pretability, and value as information used for self -improvement 
and promotion purposes. Faculty, in general, regarded the evaluative 
information to be more creditable, useful and accurate for their own 
self improvement than for promction purposes. Faculty also desired 
lore than one type of evaluative information regardless of the 
purpose of evaluation. (Author/BL) 
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Abstract 

The porpoae of the study was to Investigate faculty perceptions of student 
evaluative Inforaatlon collected by three methods: objective questionnaire Items » 
opea-*ended question, and group Interviews* Faculty rated three simulated 
evaluation reports on their potential for accuracy, trustworthiness, useful^ 
ness, compreheaalveness, bellevabUlty, InterpretablUty, and value as Informa- 
tion used for self^-lmproveaent and promotion purposes • Faculty, In general, 
regarded the evaluative Inforaatlon to be more credible, usicsful asd accurate 
for their own self improvement than for promotion purposes. Faculty also 
deslnsd more than one type of evaluative Information regardxeao of the t'urpose 
of evaluation. 
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Faculty Perceptions of the Quality and Usefulness 
of Three Types of Evaluative Information 



Evaluation of the teaching competence of faculty Is now commonplace 
in higher education (Centra, 1977). A number of different methods for collecting 
evaluative information about teaching effectiveness have been advocated (Brandenburg, 
Braskanp, Ory, 1980), including peer evaluations, self evaluations, and student 
group interviews as veil as the most popular method of student rating scales. 
The research on the iLJtl ibility and validity of student ratings has been 
voluanous (e.g., Feldaa>i, "'-977; Centra, 1980). Cohen (Note 1) in a meta- 
analysis of the relatlx^nship between student ratings and student achievement 
concluded that ratings of overall teaching were correlated with achievement 
(average r • .^7), but were not as highly correlated with specific teachings 
such as rapport, interaction, feedback and evaluation. The convergent validity 
of differing types of valuation information based on student group interviews , 
student objective rat... items, written conments to open-ended questions has been 
demonstrated if the purpose is to assess overall teaching effectiveness (Ory, 
Braskamp, and Pieper, 1980; Braskamp, Ory, and Pleper, in press). There is also 
sufficient convergent validity among evaloatior^ by peers, self, and students 
(e.g., Braskamp, CaxiUey, Costin, 1979) to warrant their use. 

However, little is known about the credit ility, trustworthiness, and 
usefulness of differing types of evaluative information from the perspective 
of the primary users > the faculty. The purpose of this study was to investigate 
faculty perceptions of student evaluative Information collected by three methods-** 
objective questionnaire items, written comments to open**ended questions, a:id 
group interviews. The major three questions asked in this study V7ere: 

1. Do faculty similarly rate the quality and usefulness of Ixiformatlon 
if the purpose is for promotion or for self Improvement? 

2. Do faculty differentially rate the quality and usefulness of the 
three types of information when it is to be used for promotion? 

3. Do faculty differentially rate the quality and usefulness of the 
three types of Information when it is to be used for self improvement? 

The evaluative information, based on three different methods of data collection, 

can be viewed as a message to a user. The conceptual framework, "Who says what 

to whoa with what effects** used extensively in communication research was employed 

to interpret the results of this study. The "who" were students; the "how" was a 

written report, "to whom" were faculty receiving the evaluative infcrution, and 



the "effects'' Included various perceptions of the faculty about the accuracy, 
trustworthiness, comprehensiveness, believability, interpretability of the 
information and the usefuln^.ss of the information for their own self improvement 
as teachers and for their promotion. In this simulation experiment, the message 
on instructor competency varied by ^ format of the presentation of the data; 
I.e., the '-what" differed by the type of information included in the report. 

This franework has been ustd in a number of previous simulation experiments 
to investigate influences of source (who), message or content (what), and 
audiences (to whom). Educators in public shoot 3 regarded evaluation reports 
without data and jargon as more believable, pra: :ical, logical, and useful 
than reports with jargon and data (Thompson, Brown, Fergason, Note 3). 
Jargon-free but data-supported reports on testing were rated the least difficult 
by another group of educators (Brown, Braskamp, and Newman, 1978). Educators 
also expressed greater agreemnt with the evaluator's recommendations about the 
program evaluation if the report included graphs and percentages but not 
statistical inference statements by the evaluator (Brown and Newman, 1980). 

Simulated Evaluation Repcrts 

Three evaluation reports about an instructor and course i^re written from 
simulated data "collected" by three different mechods: objective questionnaire 
items, open-ended questions, and group interviews. The objective items were 
from the Instructor and Course Evaluation System (ICES) student rating 
questionnaires (Office of Instructional Resources, Note 2). The ICES is a 
cafeteria-* type student rating system that permits each instructor to select 
up to 23 items to be used on the questionnaire. The first 3 items on all 
student questionnaires are global items: "Rate the course content," "Rate 
the Instructor,'' and 'Rate the course in general." Students indicate their 
rating on these 3 items on a 6-point scale, with anchor points of '*poor ' and 
"excellent*" The open-ended questions were the 4 standard questions printed 
on Side Two of the ICES questionnaire . The items are "Comment on the strengths 
and weaknesses of the instructor.' "What are the beneficial aspects of the course 
"What suggestions do you have for course Improvement? ' and ' Comment on the grading 
procedures." 

Simulated responses to the objective questionnaire items were summarized 
in a computer-generated ICES "Instructor Report." Presented in the report 
are descriptive statistics of student responses to each of the three global 
and 23 Instructor selected items. A comparison of the instructor's global 



lt€ua ratings with the normative ratings of the Instructor's department 
and of the university are also provided. The evaluation report developed 
from sloBulated written conaients to open-ended questions Included 4 hand-wrltten 
samples of Side Two of the ICES questionnaire. The comments were written to 
present a general Impression of Instructor and course quality similar to the 
one depicted by the computerized Instructor Report, Both positive and negative 
coonents about the course and Instructor were Included, 

The third evaluation report included a description and results of a 
simulated student group Interview conducted by staff members from the campus 
Office of Instructional Resources. The group interview process described in 
the slaulatlon involved a pre- and post-interview session with the Interviewer 
and the Instructor. In the pre-lntervlew session the instructor dlscucaed 
teaching philosophy, course objectives and special concems that can be 
discussed in the Interview* Following the interview a 1-2 page summary report 
of student conments was written and discussed with the instructor in an Informa] 
session. In the simulation the faculty read a one page summary. Again, tha 
general impression of instructor and course quality presented by the group 
interview report was similar to that communicated by the other two evaluation 
reports. Both strengths and x^eaknesses of the instructor and course were discussed . 

Five judges independently read each of the three evaluation reports and 
rated the comparability of content presented in each report. Each of the judges, 
staff members of the campus Office of Instructional Resources, rated the conrcnt 
highly comparable across reports. 

Data Collection 

Three groups of 50 randomly selected university faculty members received 
one of the three simulated evaluation reports during the 1980 spring semester* 
Faculty were equally represented from each academic rank and discipline areas 
of agriculture, architecture, business, chemistry-biology, classics, education, 
fine arts, math, physical sciences and social sciences. The number of faculty 
returning the rating forms for each type of evaluation report were 25 (Objective 
Item Responses), 24 (Group Interview), and 22 (Written Comments). 

Faculty were asked to rate the given re; on seven dimensions: accuracy, 
tnistworthlness, usefulness, Interpretabilit;,. comprehensiveness, bellevabillty, 
and value as information. They rated the report on each dimension twice — first 
for its use for self improvement and second for promotion purposes. For each 
item the faculty indicated on a 5-point continuum from high (»1) to no amount 



(•5) their rating on each dimension. Faculty were also asked to Indicate If 
they would uae the evaluation report for either purpose. A final question 
asked the faculty to recotnmcnd other types of evaluative Information which 
they would like to receive In addition to the information presented In the 
given evaluation report. 

Data Analysis 

To analyze differences Ir acuity perceptions, a 3 x 2 laultlvarlate 
analysis of variance (MANOVA) was computed on the seven scale lt€*^^ with 
type of evaluation report repeated across purpose. Post hoc univariate analyses 
of variance were computed to interpret significant MANOVA effects. Reported 
faculty use of the different evaluation reports was analyzed through a 1 x 3 
ANOVA design using the F test statistic. An alpha of .05 was used as the level 
of statistical significance. 

Results 

The means and standard deviations for each of the first 7 scale items 
are presented in Table 1. ResuiLs of the MANOVA presented in Table 2 revealed 
significant (p < .01) differences in faculty ratings due to the purpose of the 
report and to the type of report for particular purposes. The univariate 
analyses of variance presented in Table 2 revealed that differences due to 
purpose of report were statistically significant ror 6 of the 7 items. Faculty 



Insert Tables 1 and 2 

generally Ixidlcated that reports were potentially more accurate, trustworthy, 
useful, comprehensive, believable, and valuable when used for self-improvement 
rather than pro!ZK)tion purposes. Faculty alFo rated the group interview 
information as most comprehensive, followed in order by objective item responses 
and written student conments, regardless of the purpose of information. 

However, each type of report did not receive the same relative ratings 
of trustworthiness and bellevability for both purposes. A statistically 
significant interaction of purpose by report type was obtained for items 2 
and 6. Aa shown In figures 1 and 2, these Interactions Illustrate disordlnate 



Insert Figures 1 ar.d 2 
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Interactions, i.e., the rank order of the mean ratings of each type is c^ot 
identical for both purposes. Of the three types of evaluative information, 
the ranking of the written student conments was first for self iraprovem-r^nt 
but third for promotion on trustworthiness and believability . 

Faculty use of the information in the evaluation report is summarized 
In Table 3. Reported faculty use of the information was not significantly 
different for type of evaluation report (F [2,6?"" - 1.43, p < .30). 
Approximately 67% of the faculty were willing to use the information regardless 
of typK of evaluation report. 



Insert Table 3 

Faculty, when asked if they also vanted to receive the other types of 
evaluative information, generally desired more information. Of faculty 
receiving written comments information, 63*2Z and 54.52 i«)uld have also 
appreciated receiving objective item resr>onses and group interview summaries, 
respectively. Approximately eight percent of the faculty responding to the 
group interview report asked for objective item responses and written comments 
about their perform.- e. Finally, 63% and 40% of the faculty rating the 
objective item response report also wanted to receive written comments and 
group interview results^ respectively. Ad proximately one-third of all 
respondents also requested that some form of peer review be used to evaluate 
their performance. 

Discussion 

Based on this simulation experiment, tae tyoe of information presented — 
student written comments, computer printout of student response to fixed 
alternative questicnnaire items, and written suinmary of student group interview — 
does influence t>»3 audience's (faculty) rating of the technical quality and 
credibility of tae information. However the conceptual framework, 'who says 
what how to %ahom with what effects*' needs to be expanded to ' unJer what 
conditions who says what how to whom with what effects. ' The conditions in 
this slikuiatiou refer to the purpose of the evaluation. In six of the seven 
ratings of the quuility and credibility of evaluative information, faculty in 
general regarded the information to be more credible, useful, and accurate for 
their own self impro"^^ment than for promotion purposes. 
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However, faculty differentially rated the credibility (i.e., trustworthiness 
and believability) of the three types of information depend inp, on the purpose, 
Faculty regarded student written conimrnts as less credible than student responses 
to objective comments when the purpose was promotion but rated written coinnents 
as more credible when the purpose w-.s self improveir.ent , Perhaps the non- 
standardized, unique and personal written concnents by students are perceived as 
too subjective for important personnel decisions. However, this highly 
Idiosyncratic type of information abo;^r a particular course is viewed as useful 
diagnostic information for making course changes* 

Finally, the faculty desirec -ore than one type of evalur.rive information 
regardless of tne purpose of evaluation. One third of the faculty also 
desired peer reviews^ even though peer reviews are seldom done on this campus. 
Faculty in this simulation have supported the advice giveti b; many in faculty 
evaluation (e.g., Doyle, 1975, Centra, 1980, Brandenburg, Braskamp, and Ory, 1980) 
that evaluation information about teaching competence should be collected from 
many different sources (e.g., peers, students) using a variety of differing 
methods of data collection. 
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Table 1 

Scale Item Means and Standard Deviations 



Type of Report Self -Improvement Purposes Promotion Purposes 

Y SD Y ST) 

Item ifl: Hov accurate? 

Group interview (n - 24) 2.56^ .73 2.P4 1.06 

Objective Item responses (n - 25) 2.81 1.29 3.14 1.32 

Written comments (n - 22) 2.53 .99 3.47 1.30 

Item #2: Hov trustworthy? 

Group interview 2.63 .62 2.38 1.03 

Objecrlve Item responses 3.05 1.12 3,19 1.17 

Written comnents 2.60 1.12 3.67 1.13 

Item y3: How useful? 

Group interview 2.19 1.11 2.75 1.13 

Objective item responses 2.86 1.46 3.10 1.26 

Written comments 2.60 1.13 3.40 1.35 

Item #4; How difficult to Interpret ? 

Group interview 3.19 1.05 3.25 1.07 

Objective item responses 2.95 1.32 2.31 1.33 

VTrltten coTnnents 3.60 1.12 2.93 1.67 

Item #5: How comprehensive? 

Group interview 3.36 1.06 3.33 1.03 

Objective item responses 3.72 1.10 3.81 1.17 

Written comments ^-13 .33 4.33 .72 

Item #6: How believable? 

— Group interview 2.69 1.08 3.06 1.06 

Objective item responses 2.95 1.16 3.10 1.09 

Written comments 2.47 .99 3.33 1.35 

Item #7; How valuable? 

Group interview 2.38 1.31 2.9* 1.24 

Objective item responses 2.91 1.45 3.14 1.39 

Written comments 2.73 1.03 3.33 1.23 

gj^(^>w scale values indicate high perceived accuracy, trustworthiness, usefulness, etc. 

™" 10 
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Table 2 

IIANOVA & AMOVA Summary Tables: Iteras 1-7 



Source of Variatiou <;S jf vj<^ 



MANOtfA 

Type of Report (T) 35.26 14.497 2.52 1.12 

Between Error 800.36 2.24 

Purpose of Report (P) 36. 3S 7,497 5.27 15.55* 

Purpose X Type (PT) 12.70 14,497 .91 2.63* 

Within Error 120.92 .34 

ANOVAS 

Item #1: Hov accurate? 

1.25 2 .62 .27 

ue^ween Error il4.48 68 2.34 

P 7.62 1 7.62 23.33** 

^ 1.G2 2 .91 2.84 

Within Error 15.63 68 .32 

:em #2: Hov trustworthy? 

T 3.12 2 1.56 .80 

Between Error 55.37 68 1.95 

p 6.02 1 6.0.'' 19.36** 

?T 4.16 2 2 OS 6.63** 

Within Error 15.25 63 .31 

Item if3; How useful? 

T 5.94 2 2.97 1.09 

Between Error 133.95 68 2.73 

P 7.25 1 7.25 15.39** 

PT 1.43 2 .71 1.51 

Within Error 23.07 63 .47 

Item #4: Hov difficult to interpret? 

T 3.30 2 1.65 .53 

Between Error 139.74 63 2.85 

P 1.58 1 1.53 3.79 

PT 2.19 2 1.10 2.63 

O 

ERLC within 2.Tror 20.42 ^ 68 .42 
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Table 2, continued. 



Source of Variation 



df 



'IS 



Item if 5: How comprehensive? 

T 16.01 

Between Error 92.45 

P l.Oi 

PT .22 

Within Error 8.r;2 

Item #6: How believable? 

T .4d 

Between Error 103.13 

P 5.42 

PT 2.32 

Within Error 16.03 

Item #7: JHow valuable? 

T 3.06 

Between Error 149.66 

P 5.55 

PT .74 

Within Error 15.67 



2 
68 
1 
2 
68 

2 
68 
1 
2 
68 

2 
68 
1 

2 

68 



R.OO 

i.r,9 

1.05 

.11 

. 24 
2.21 
5.42 
1.16 

.3^ 

1.53 
3.05 
5.55 
.37 
. 32 



. 24* 

5.80* 
."0 



. 11 

16.53** 
3. 54* 



. 50 

17.35* 
1.16 



*p < .05 
**p < .01 
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Table 3 

Faculty Use and Preference for Student Use of Information 



Objective Item Written Group 

Item Responses Responses (n«25) Comments (n«22) Interview (n«24) 





f 




2 


f 




Z 


f 


% 


Item #8: Would you use the 


information? 












Definitely would (1) 


7 




28.0 


7 




31.8 


10 


41.7 


(2) 


9 




36.0 


7 




31.8 


8 


33.3 


(3) 


4 




16.0 


4 




18.2 


5 


on 

zu* o 


(4) 


1 




4.0 


2 




9.0 


1 


4.2 


Definitely would not (5) 


4 




16.0 


2 




9.0 


0 


0 






X 


- 2.44 




X 


- 2.32 


X 


- 1.88 






SD 


- 1.39 




SD 


- 1.29 


SD 


- .90 


Item #9: Should students have 


access to the 


information to help thea in 


course 


selection? 


















Definitely should (1) 


8 




32.0 


4 




18.2 


9 


37.5 


(2) 


2 




8.0 


3 




13.6 


6 


25.0 


(3) 


5 




20.0 


6 




27.3 


4 


16.7 


(A) 


4 




16.0 


6 




27.3 


3 


12.5 


Definitely should not (5) 


6 




24.0 


3 




13.6 


2 


8.3 






X 


= 2.92 




X 


- 3.05 


X 


- 2.29 






SD 


= 1.61 




SD 


- 1.33 


SD 


" 1.33 
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Hot at all 5 
Trustworthy 




Figure 1* Purpose x type of report Interaction for scale Item #2: Trustworthiness 



Not at &1I 5 
Belie vablv^ 



i- 



3 



Vcty, Believable 1 




Written cotnments 
^ ^ Objective iteins 
Group interviev 



Self-Improvement Promotion 
Purpose 



figure 2. Purpose X type of report Interaction for seals item #6: Believability 



ERIC 



15 



